Project Outline¶

  1. Data Loading and Preprocessing

    1. Load the dataset.
    2. Exploratory Data Analysis:
      • Feature histograms.
      • Train-Test Split (to avoid leakage).
      • Check for outliers (and optional removal).
      • Scaling (Standardization).
      • Correlation Matrix and Multicollinearity Check:
        • Remove highly correlated features.
    3. Prepare final features for analysis.
  2. Linear Algorithms

    1. Perceptron
      • Train with 5-fold cross-validation (CV).
      • Hyperparameter tuning using CV.
      • Evaluate the best-tuned model on the test set.
    2. Pegasos SVM
      • Train with 5-fold cross-validation.
      • Evaluate the model on the test set.
    3. Logistic Regression
      • Train with 5-fold cross-validation.
      • Evaluate the model on the test set.
    4. Comparison of Results
      • Summarize and compare accuracies across Perceptron, Pegasos SVM, and Logistic Regression.
  3. Polynomial Feature Expansion

    1. Expand features to degree 2 (include squared terms and interactions).
    2. Retrain and evaluate:
      • Perceptron with polynomial expansion.
      • Pegasos SVM with polynomial expansion.
      • Logistic Regression with polynomial expansion.
    3. Summarize and compare results:
      • Linear vs. Polynomial-expanded features.
      • Visualize results.
  4. Kernelized Methods

    1. Kernelized Perceptron
      • Implement and evaluate with:
        • Gaussian Kernel.
        • Polynomial Kernel.
      • Compare test set accuracies.
    2. Kernelized Pegasos SVM
      • Implement and evaluate with:
        • Gaussian Kernel.
        • Polynomial Kernel.
      • Compare test set accuracies.
  5. Final Comparison

    1. Summarize final results:
      • Linear models.
      • Polynomial-expanded models.
      • Kernelized models.

Kernelized Linear Classification¶

Initial imports + functions¶

Basic libraries + custom functions (ffunctions.py):

In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Uncomment the following line if you want that plots appear inline in a real Jupyter notebook:
# %matplotlib inline

from ffunctions import (
    load_data,
    train_test_split,
    detect_outliers_zscore,
    remove_outliers,
    standard_scaler_fit,
    standard_scaler_transform,
    check_high_correlation,
    k_fold_cross_validation,
    perceptron_classifier_func,
    pegasos_classifier_func,
    logistic_regression_classifier_func,
    perceptron_classifier_func_eta,
    perceptron_train,
    perceptron_predict,
    pegasos_train,
    pegasos_predict,
    logistic_regression_train,
    logistic_regression_predict,
    polynomial_feature_expansion,
    gaussian_kernel,
    polynomial_kernel,
    kernelized_perceptron_classifier_func,
    kernelized_perceptron_train,
    kernelized_perceptron_predict,
    kernelized_pegasos_classifier_func,
    kernelized_pegasos_train,
    kernelized_pegasos_predict
)

best_hparams_table = {
    "Perceptron": {},
    "Pegasos": {},
    "Logistic": {},
    "PerceptronPoly": {},
    "PegasosPoly": {},
    "LogisticPoly": {},
    "KernelPerceptronGauss": {},
    "KernelPerceptronPoly": {},
    "KernelPegasosGauss": {},
    "KernelPegasosPoly": {}
}

Data loading:

In [2]:
csv_path = "/Users/michelecoaro/Documents/GitHub/KLC/klc_project/dataset.csv"
df = load_data(csv_path)
df.head()
Out[2]:
x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 y
0 1.205492 5.823226 98.837539 -1.075852 0.999205 0.911543 3.623558 -1.720267 -0.346191 -54.708330 -1
1 1.692571 -0.887019 100.901276 -0.595548 0.177550 -0.915495 4.320264 0.907834 3.126815 -56.397484 -1
2 4.289320 1.416843 100.784735 -2.897154 -0.066972 -0.786173 2.093003 1.336237 2.183829 -56.197728 1
3 0.542420 -1.010095 100.015580 -3.070705 0.088324 -0.242669 0.767942 -0.284683 -2.104145 -55.794045 1
4 2.431765 -1.224177 100.709237 -0.102892 0.699031 -0.656315 -0.911784 -1.355873 1.537825 -55.917863 -1

Exploratory Data Analysis¶

Understanding feature locations and shape

In [3]:
feature_cols = [f"x{i}" for i in range(1, 11)]
df[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features", fontsize=16)
plt.show()
No description has been provided for this image
In [4]:
#pairplot of features with different colors for each feature in the pair
import seaborn as sns
sns.pairplot(df, hue='y', diag_kind='hist')
plt.suptitle("Pairplot of Numerical Features", fontsize=16)
plt.show()
No description has been provided for this image

Train - Test split as first data leakage countermeasure:

In [5]:
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

print("Train set shape:", train_df.shape)
print("Test set shape:", test_df.shape)
Train set shape: (7788, 11)
Test set shape: (1947, 11)

Checking for outliers in training set

In [6]:
outlier_indices = detect_outliers_zscore(train_df, feature_cols, z_thresh=3.0)
print(f"Number of outliers detected in training set: {len(outlier_indices)}")
Number of outliers detected in training set: 82

Outliers removal:

In [7]:
train_df_no_outliers = remove_outliers(train_df, outlier_indices)
print("Train set (no outliers) shape:", train_df_no_outliers.shape)
Train set (no outliers) shape: (7706, 11)

Applying consistent scaling

In [8]:
means, stds = standard_scaler_fit(train_df_no_outliers, feature_cols)
train_scaled = standard_scaler_transform(train_df_no_outliers, feature_cols, means, stds)
test_scaled = standard_scaler_transform(test_df, feature_cols, means, stds)
In [9]:
# histogram plot for numerical variables after scaling
train_scaled[feature_cols].hist(bins=30, figsize=(15, 10))
plt.suptitle("Histograms of Numerical Features (After Scaling)", fontsize=16)
plt.show()
No description has been provided for this image
In [10]:
#side by side plot of features before and after scaling

fig, axs = plt.subplots(2, 5, figsize=(20, 8))
fig.suptitle("Side-by-Side Histograms of Numerical Features Before and After Scaling", fontsize=16)

for i, col in enumerate(feature_cols):
    train_df[col].hist(bins=30, ax=axs[i // 5, i % 5])
    axs[i // 5, i % 5].set_title(f"Before Scaling: {col}")

fig, axs = plt.subplots(2, 5, figsize=(20, 8))
for i, col in enumerate(feature_cols):
    train_scaled[col].hist(bins=30, ax=axs[i // 5, i % 5])
    axs[i // 5, i % 5].set_title(f"After Scaling: {col}")

plt.show()
No description has been provided for this image
No description has been provided for this image

Correlation Matrix

In [11]:
# 5. Correlation Matrix (for numerical columns)
corr_matrix = train_scaled[feature_cols].corr()
plt.figure(figsize=(8, 6))
plt.imshow(corr_matrix, cmap='coolwarm', aspect='auto')
plt.colorbar(label='Correlation Coefficient')
plt.title('Correlation Matrix')
plt.xticks(range(len(corr_matrix.columns)), corr_matrix.columns, rotation=45)
plt.yticks(range(len(corr_matrix.columns)), corr_matrix.columns)
plt.tight_layout()
plt.show()
No description has been provided for this image

Multicollinearity Check

In [12]:
high_corr_pairs = check_high_correlation(train_df_no_outliers, feature_cols, corr_threshold=0.9)
if high_corr_pairs:
    print("High correlation pairs detected (above threshold):")
    for pair in high_corr_pairs:
        print(f"  {pair[0]} and {pair[1]} with correlation = {pair[2]:.3f}")
else:
    print("No pairs of features exceed the correlation threshold.")
High correlation pairs detected (above threshold):
  x3 and x6 with correlation = 0.990
  x3 and x10 with correlation = 0.980
  x6 and x10 with correlation = 0.990

Removing unwanted columns

In [13]:
for col in ["x3", "x6", "x10"]:
    corr = train_df_no_outliers[col].corr(train_df_no_outliers["y"])
    print(col, corr)

final_features = [f for f in feature_cols if f not in ["x3", "x6"]]
print("\nDropping x3, x6 from the final feature set...")
print("Final features:", final_features)
x3 0.051673811986996
x6 -0.05343499561215009
x10 -0.05288351228543469

Dropping x3, x6 from the final feature set...
Final features: ['x1', 'x2', 'x4', 'x5', 'x7', 'x8', 'x9', 'x10']

Algorithms Implementation¶

Cross-validation¶

In [14]:
target_col = "y"

# Example: Perceptron with 5 epochs
accuracy_perc = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
        tr_fold, vl_fold, f, t, epochs=5
    )
)
print(f"Perceptron 5-fold CV Accuracy: {accuracy_perc:.3f}")

# Example: Pegasos
accuracy_pegasos = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
        tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5
    )
)
print(f"Pegasos 5-fold CV Accuracy: {accuracy_pegasos:.3f}")

# Example: Logistic Regression
accuracy_logreg = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
        tr_fold, vl_fold, f, t, lambda_param=0.01, epochs=5, eta=1.0
    )
)
print(f"Logistic Regression 5-fold CV Accuracy: {accuracy_logreg:.3f}")
Perceptron 5-fold CV Accuracy: 0.585
Pegasos 5-fold CV Accuracy: 0.723
Logistic Regression 5-fold CV Accuracy: 0.718

Perceptron¶

Hyperparameter Tuning¶

In [15]:
target_col = "y"
epochs_list = [5, 10, 20, 50]
learning_rates = [0.01, 0.1, 1.0]

tuning_results_perc = {}

for ep in epochs_list:
    for eta in learning_rates:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
                tr, vl, f, t, epochs=ep, eta=eta
            )
        )
        tuning_results_perc[(ep, eta)] = acc

best_params_perc = max(tuning_results_perc, key=tuning_results_perc.get)
best_epochs_perc, best_eta_perc = best_params_perc
best_acc_perc = tuning_results_perc[best_params_perc]

print("Perceptron best hyperparams =>", best_params_perc, "CV Accuracy =", best_acc_perc)

# Save in best_hparams_table
best_hparams_table["Perceptron"]["epochs"] = best_epochs_perc
best_hparams_table["Perceptron"]["eta"] = best_eta_perc
best_hparams_table["Perceptron"]["cv_accuracy"] = best_acc_perc

# Train final model on entire train set with best hyperparams
from ffunctions import perceptron_train_eta, perceptron_predict

theta_perc, theta0_perc = perceptron_train_eta(
    train_scaled, final_features, target_col,
    epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_perc = perceptron_predict(test_scaled, final_features, theta_perc, theta0_perc)
test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Perceptron final test accuracy (best hyperparams): {test_acc_perc:.3f}")
Perceptron best hyperparams => (10, 0.1) CV Accuracy = 0.6466382349797284
Perceptron final test accuracy (best hyperparams): 0.601
In [16]:
# Compare default and tuned Perceptron across feature subsets
subsets = {
    "Features": feature_cols}

comparison_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating subset: {subset_name}")
    
    # Default Perceptron
    accuracy_default_perc = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func(
            tr_fold, vl_fold, f, t, epochs=5  # Default epochs
        )
    )
    
    # Tuned Perceptron
    accuracy_tuned_perc = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: perceptron_classifier_func_eta(
            tr_fold, vl_fold, f, t, epochs=best_epochs_perc, eta=best_eta_perc
        )
    )
    
    comparison_results[subset_name] = {
        "Default Perceptron": accuracy_default_perc,
        "Tuned Perceptron": accuracy_tuned_perc,
    }
    
    print(f"  Default Perceptron Accuracy: {accuracy_default_perc:.3f}")
    print(f"  Tuned Perceptron Accuracy: {accuracy_tuned_perc:.3f}")
Evaluating subset: Features
  Default Perceptron Accuracy: 0.624
  Tuned Perceptron Accuracy: 0.627

Prediction on test dataset

In [17]:
theta_perc, theta0_perc = perceptron_train(
    train_scaled,
    features=final_features,
    target=target_col,
    epochs=5
)
test_preds_perc = perceptron_predict(
    test_scaled,
    features=final_features,
    theta=theta_perc,
    theta_0=theta0_perc
)

test_acc_perc = np.mean(test_preds_perc == test_scaled[target_col])
print(f"Linear Perceptron final test accuracy: {test_acc_perc:.3f}")
Linear Perceptron final test accuracy: 0.670

Pegasos SVM¶

Hyperparameter Tuning

In [18]:
lambda_values_pega = [0.001, 0.01, 0.1]
epochs_values_pega = [5, 10, 20]

tuning_results_pegasos = {}
for lam in lambda_values_pega:
    for e in epochs_values_pega:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
                tr, vl, f, t, lambda_param=lam, epochs=e
            )
        )
        tuning_results_pegasos[(lam, e)] = acc

best_lambda_pega, best_epochs_pega = max(tuning_results_pegasos, key=tuning_results_pegasos.get)
best_acc_pega_cv = tuning_results_pegasos[(best_lambda_pega, best_epochs_pega)]

print("Pegasos best hyperparams =>", (best_lambda_pega, best_epochs_pega), "CV =", best_acc_pega_cv)

best_hparams_table["Pegasos"]["lambda"] = best_lambda_pega
best_hparams_table["Pegasos"]["epochs"] = best_epochs_pega
best_hparams_table["Pegasos"]["cv_accuracy"] = best_acc_pega_cv

# Train final Pegasos
theta_pega, theta0_pega = pegasos_train(
    train_scaled, final_features, target_col,
    lambda_param=best_lambda_pega, epochs=best_epochs_pega
)
test_preds_pega = pegasos_predict(test_scaled, final_features, theta_pega, theta0_pega)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy (best hyperparams): {test_acc_pega:.3f}")
Pegasos best hyperparams => (0.001, 20) CV = 0.7245005727579326
Pegasos final test accuracy (best hyperparams): 0.741

Comparison of different subsets¶

In [19]:
# Use the best hyperparameters found during tuning
lambda_param_pega = best_lambda_pega
epochs_pega = best_epochs_pega
# Compare Pegasos SVM across feature subsets
pegasos_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating Pegasos SVM on subset: {subset_name}")
    
    accuracy_pegasos = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: pegasos_classifier_func(
            tr_fold, vl_fold, f, t, lambda_param=lambda_param_pega, epochs=epochs_pega
        )
    )
    
    pegasos_results[subset_name] = accuracy_pegasos
    print(f"  Pegasos SVM Accuracy: {accuracy_pegasos:.3f}")
Evaluating Pegasos SVM on subset: Features
  Pegasos SVM Accuracy: 0.711

Measring accuracy of prediction¶

By testing on test set

In [20]:
theta_pega, theta0_pega = pegasos_train(
    train_scaled,
    features=final_features,
    target=target_col,
    epochs=5
)
test_preds_pega = pegasos_predict(
    test_scaled,
    features=final_features,
    theta=theta_pega,
    theta_0=theta0_pega
)
test_acc_pega = np.mean(test_preds_pega == test_scaled[target_col])
print(f"Pegasos final test accuracy: {test_acc_pega:.3f}")
Pegasos final test accuracy: 0.733

Logistic Regression¶

Hyperparameter tuning

In [21]:
lambda_list_logreg = [0.001, 0.01, 0.1]
epochs_list_logreg = [5, 10]
eta_list_logreg = [0.01, 0.1, 1.0]

tuning_results_logreg = {}
for lam in lambda_list_logreg:
    for ep in epochs_list_logreg:
        for eta in eta_list_logreg:
            acc = k_fold_cross_validation(
                df=train_scaled,
                features=final_features,
                target=target_col,
                k=5,
                random_state=42,
                classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
                    tr, vl, f, t,
                    lambda_param=lam,
                    epochs=ep,
                    eta=eta
                )
            )
            tuning_results_logreg[(lam, ep, eta)] = acc

best_lam_logreg, best_ep_logreg, best_eta_logreg = max(tuning_results_logreg, key=tuning_results_logreg.get)
best_acc_logreg_cv = tuning_results_logreg[(best_lam_logreg, best_ep_logreg, best_eta_logreg)]

print("Logistic best hyperparams =>", (best_lam_logreg, best_ep_logreg, best_eta_logreg), "CV =", best_acc_logreg_cv)

best_hparams_table["Logistic"]["lambda"] = best_lam_logreg
best_hparams_table["Logistic"]["epochs"] = best_ep_logreg
best_hparams_table["Logistic"]["eta"] = best_eta_logreg
best_hparams_table["Logistic"]["cv_accuracy"] = best_acc_logreg_cv

# Train final logistic
theta_lr, theta0_lr = logistic_regression_train(
    train_scaled, final_features, target_col,
    lambda_param=best_lam_logreg,
    epochs=best_ep_logreg,
    eta=best_eta_logreg
)
test_preds_lr = logistic_regression_predict(test_scaled, final_features, theta_lr, theta0_lr)
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic final test accuracy (best hyperparams): {test_acc_lr:.3f}")
Logistic best hyperparams => (0.1, 10, 1.0) CV = 0.7173635291652042
Logistic final test accuracy (best hyperparams): 0.729
In [22]:
# Define parameters for Logistic Regression
lambda_param_logreg = best_lam_logreg
epochs_logreg = best_ep_logreg
eta_logreg = best_eta_logreg

# Compare Logistic Regression across feature subsets
logistic_results = {}

for subset_name, subset_features in subsets.items():
    print(f"\nEvaluating Logistic Regression on subset: {subset_name}")
    
    accuracy_logistic = k_fold_cross_validation(
        train_scaled,
        features=subset_features,
        target='y',
        k=5,
        random_state=42,
        classifier_func=lambda tr_fold, vl_fold, f, t: logistic_regression_classifier_func(
            tr_fold, vl_fold, f, t, lambda_param=best_lam_logreg, epochs=best_ep_logreg, eta=best_eta_logreg
        )
    )
    logistic_results[subset_name] = accuracy_logistic
    print(f"  Logistic Regression Accuracy: {accuracy_logistic:.3f}")
Evaluating Logistic Regression on subset: Features
  Logistic Regression Accuracy: 0.714
In [23]:
# Train the Logistic Regression model
theta_lr, theta0_lr = logistic_regression_train(
    train_scaled,
    features=final_features,
    target=target_col,
    lambda_param=best_lam_logreg,  # Regularization parameter
    epochs=best_ep_logreg,
    eta=best_eta_logreg             # Learning rate
)

# Predict on the test set
test_preds_lr = logistic_regression_predict(
    test_scaled,
    features=final_features,
    theta=theta_lr,
    theta_0=theta0_lr
)

# Compute accuracy
test_acc_lr = np.mean(test_preds_lr == test_scaled[target_col])
print(f"Logistic Regression final test accuracy: {test_acc_lr:.3f}")
Logistic Regression final test accuracy: 0.734

Comparison¶

In [24]:
# Combine results into a DataFrame
comparison_with_logistic = pd.DataFrame({
    "Default Perceptron Loss": [comparison_results[name]["Default Perceptron"] for name in subsets.keys()],
    "Tuned Perceptron Accuracy": [comparison_results[name]["Tuned Perceptron"] for name in subsets.keys()],
    "Pegasos SVM Accuracy": [pegasos_results[name] for name in subsets.keys()],
    "Logistic Regression Accuracy": [logistic_results[name] for name in subsets.keys()]
}, index=list(subsets.keys()))

# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on training set:")
display(comparison_with_logistic)

# Comparison table on test set
comparison_test = pd.DataFrame({
    "Perceptron (Default)": [test_acc_perc],
    "Perceptron (Tuned)": [comparison_results["Features"]["Tuned Perceptron"]],
    "Pegasos SVM": [test_acc_pega],
    "Logistic Regression": [test_acc_lr]
}, index=["Test Set"])

# Display the comparison table
print("\nComparison of Perceptron, Pegasos, and Logistic Regression on test set:")
display(comparison_test)
Comparison of Perceptron, Pegasos, and Logistic Regression on training set:
Default Perceptron Loss Tuned Perceptron Accuracy Pegasos SVM Accuracy Logistic Regression Accuracy
Features 0.62392 0.626535 0.711005 0.714379
Comparison of Perceptron, Pegasos, and Logistic Regression on test set:
Perceptron (Default) Perceptron (Tuned) Pegasos SVM Logistic Regression
Test Set 0.670262 0.626535 0.732922 0.73395

Polynomial feature expansion¶

In [25]:
# Expand the scaled training data (train_scaled) and test data (test_scaled)
# using the final_features (excluding x3, x6). Then rescale the newly created polynomial columns.

# 1) Expand polynomial features on the *scaled* train and test data.
train_poly_unscaled = polynomial_feature_expansion(train_scaled, final_features, degree=2, include_bias=False)
test_poly_unscaled = polynomial_feature_expansion(test_scaled, final_features, degree=2, include_bias=False)

# Identify new polynomial columns (excluding 'y')
expanded_feature_cols = [col for col in train_poly_unscaled.columns if col != 'y']

print("Polynomial expansion complete.")
print("Number of expanded features:", len(expanded_feature_cols))
print("Sample of expanded features:", expanded_feature_cols[:10])
Polynomial expansion complete.
Number of expanded features: 46
Sample of expanded features: ['x1', 'x1^2', 'x2', 'x2^2', 'x4', 'x4^2', 'x5', 'x5^2', 'x7', 'x7^2']
In [26]:
# 2) Rescale the newly expanded features
means_poly, stds_poly = standard_scaler_fit(train_poly_unscaled, expanded_feature_cols)
train_poly = standard_scaler_transform(train_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)
test_poly = standard_scaler_transform(test_poly_unscaled, expanded_feature_cols, means_poly, stds_poly)

k = 5
target_col = "y"

Perceptron algorithm with polynomial expansion¶

In [27]:
acc_perc_poly = k_fold_cross_validation(
    df=train_poly,
    features=expanded_feature_cols,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr, vl, f, t: perceptron_classifier_func_eta(
        tr, vl, f, t, epochs=best_epochs_perc, eta=best_eta_perc
    )
)

theta_perc_poly, theta0_perc_poly = perceptron_train_eta(
    train_poly, expanded_feature_cols, target_col,
    epochs=best_epochs_perc, eta=best_eta_perc
)
test_preds_poly_perc = perceptron_predict(
    test_poly, expanded_feature_cols, theta_perc_poly, theta0_perc_poly
)
test_acc_poly_perc = np.mean(test_preds_poly_perc == test_poly[target_col])

print("\nPerceptron + Poly => CV=%.3f, Test=%.3f" % (acc_perc_poly, test_acc_poly_perc))
best_hparams_table["PerceptronPoly"]["epochs"] = best_epochs_perc
best_hparams_table["PerceptronPoly"]["eta"] = best_eta_perc
best_hparams_table["PerceptronPoly"]["cv_accuracy"] = acc_perc_poly
Perceptron + Poly => CV=0.926, Test=0.924

SVM with PE¶

In [28]:
lambda_list_poly = [0.001, 0.01, 0.1]
epochs_list_poly = [5, 10, 20]
tuning_results_pegasos_poly = {}

for lam in lambda_list_poly:
    for ep in epochs_list_poly:
        acc = k_fold_cross_validation(
            df=train_poly,
            features=expanded_feature_cols,
            target=target_col,
            k=5,
            random_state=42,
            classifier_func=lambda tr, vl, f, t: pegasos_classifier_func(
                tr, vl, f, t, lambda_param=lam, epochs=ep
            )
        )
        tuning_results_pegasos_poly[(lam, ep)] = acc

best_lam_pega_poly, best_ep_pega_poly = max(tuning_results_pegasos_poly, key=tuning_results_pegasos_poly.get)
acc_pega_poly = tuning_results_pegasos_poly[(best_lam_pega_poly, best_ep_pega_poly)]

print("Pegasos + Poly best hyperparams =>", (best_lam_pega_poly, best_ep_pega_poly), "CV =", acc_pega_poly)
best_hparams_table["PegasosPoly"]["lambda"] = best_lam_pega_poly
best_hparams_table["PegasosPoly"]["epochs"] = best_ep_pega_poly
best_hparams_table["PegasosPoly"]["cv_accuracy"] = acc_pega_poly

# Train final Pegasos on polynomial
theta_pega_poly, theta0_pega_poly = pegasos_train(
    train_poly, expanded_feature_cols, target_col,
    lambda_param=best_lam_pega_poly, epochs=best_ep_pega_poly
)
test_preds_pega_poly = pegasos_predict(
    test_poly, expanded_feature_cols, theta_pega_poly, theta0_pega_poly
)
test_acc_pega_poly = np.mean(test_preds_pega_poly == test_poly[target_col])
print(f"Pegasos + Poly final test accuracy: {test_acc_pega_poly:.3f}")
Pegasos + Poly best hyperparams => (0.001, 20) CV = 0.9457567516839758
Pegasos + Poly final test accuracy: 0.949

Logistic Regression with PE¶

Training set cross validation:

In [29]:
acc_logreg_poly = k_fold_cross_validation(
    df=train_poly,
    features=expanded_feature_cols,
    target=target_col,
    k=5,
    random_state=42,
    classifier_func=lambda tr, vl, f, t: logistic_regression_classifier_func(
        tr, vl, f, t,
        lambda_param=best_lam_logreg,
        epochs=best_ep_logreg,
        eta=best_eta_logreg
    )
)

theta_logreg_poly, theta0_logreg_poly = logistic_regression_train(
    train_poly, expanded_feature_cols, target_col,
    lambda_param=best_lam_logreg,
    epochs=best_ep_logreg,
    eta=best_eta_logreg
)
test_preds_logreg_poly = logistic_regression_predict(
    test_poly, expanded_feature_cols, theta_logreg_poly, theta0_logreg_poly
)
test_acc_logreg_poly = np.mean(test_preds_logreg_poly == test_poly[target_col])

print("LogReg + Poly => CV=%.3f, Test=%.3f" % (acc_logreg_poly, test_acc_logreg_poly))
best_hparams_table["LogisticPoly"]["lambda"] = best_lam_logreg
best_hparams_table["LogisticPoly"]["epochs"] = best_ep_logreg
best_hparams_table["LogisticPoly"]["eta"] = best_eta_logreg
best_hparams_table["LogisticPoly"]["cv_accuracy"] = acc_logreg_poly
LogReg + Poly => CV=0.882, Test=0.882

Results¶

In [30]:
print("\n=== Polynomial-Expanded Feature Set Accuracies ===")
print(f"Perceptron (poly) CV: {acc_perc_poly:.3f},  Test: {test_acc_poly_perc:.3f}")
print(f"Pegasos (poly)    CV: {acc_pega_poly:.3f}, Test: {test_acc_pega_poly:.3f}")
print(f"LogReg (poly)     CV: {acc_logreg_poly:.3f},  Test: {test_acc_logreg_poly:.3f}")
=== Polynomial-Expanded Feature Set Accuracies ===
Perceptron (poly) CV: 0.926,  Test: 0.924
Pegasos (poly)    CV: 0.946, Test: 0.949
LogReg (poly)     CV: 0.882,  Test: 0.882

Compare to Linear (Non-Expanded) Results¶

We already have our linear results for Perceptron, Pegasos, and Logistic from previous cells:

  • accuracy_perc, accuracy_pegasos, accuracy_logreg (5-fold on linear features).

Let's create a small summary:

In [31]:
comparison_poly = pd.DataFrame({
    "Linear": [accuracy_perc, accuracy_pegasos, accuracy_logreg],
    "Polynomial(2)": [acc_perc_poly, acc_pega_poly, acc_logreg_poly]
}, index=["Perceptron", "Pegasos", "Logistic"])

print("Comparison of Linear vs Polynomial(2) feature sets:")
display(comparison_poly)
Comparison of Linear vs Polynomial(2) feature sets:
Linear Polynomial(2)
Perceptron 0.585118 0.925513
Pegasos 0.711005 0.945757
Logistic 0.717883 0.882041

Comparison Plot¶

In [32]:
models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = comparison_poly["Linear"].values
poly_accs   = comparison_poly["Polynomial(2)"].values
x = np.arange(len(models))
width = 0.35

plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs,  width, label='Linear')
plt.bar(x + width/2, poly_accs,    width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('CV Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()
No description has been provided for this image
In [33]:
# comparison plot that shows the accuracy of the models on the test set

models = ["Perceptron", "Pegasos", "Logistic"]
linear_accs = [test_acc_perc, test_acc_pega, test_acc_lr]
poly_accs   = [test_acc_poly_perc, test_acc_pega_poly, test_acc_logreg_poly]
x = np.arange(len(models))

plt.figure(figsize=(8,4))
plt.bar(x - width/2, linear_accs,  width, label='Linear')
plt.bar(x + width/2, poly_accs,    width, label='Polynomial(2)')
plt.xticks(x, models)
plt.ylabel('Test Accuracy')
plt.title('Linear vs. Polynomial(2) Feature Expansion')
plt.ylim([0, 1])
plt.legend()
plt.tight_layout()
plt.show()
No description has been provided for this image

Weights inspection¶

In [34]:
theta_perc_poly, theta_0_perc_poly = perceptron_train(
    train_poly, 
    features = expanded_feature_cols, 
    target = target_col,
    epochs = 5
)
print(f"Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= {theta_0_perc_poly:.3f}")

weights_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_perc_poly,
}).sort_values(by="Weight", ascending=False)

# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))
Perceptron (poly) final model trained on entire train_poly. Bias (theta_0)= 28.000

Top 10 Positive Weights:
Feature Weight
27 x2*x9 108.472657
10 x8 57.616740
20 x1*x8 46.761335
12 x9 24.724363
6 x5 20.067918
0 x1 17.537309
8 x7 13.122208
44 x3 11.087068
41 x8*x9 7.061366
35 x5*x8 6.677310
Top 10 Negative Weights:
Feature Weight
9 x7^2 -3.733855
19 x1*x7 -5.428044
43 x9*x10 -5.533118
5 x4^2 -6.143356
45 x6 -6.267363
13 x9^2 -6.325809
18 x1*x5 -7.307161
17 x1*x4 -12.237412
4 x4 -21.615008
31 x4*x8 -29.930359
In [35]:
theta_pegasos_poly, theta_0_pegasos_poly = pegasos_train(
    train_poly,
    features=expanded_feature_cols,
    target=target_col,
    lambda_param=0.01,
    epochs=5
)
print(f"Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = {theta_0_pegasos_poly:.3f}")

weights_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_pegasos_poly
}).sort_values(by="Weight", ascending=False)

# Show top 10 positive weights and top 10 negative weights
print("\nTop 10 Positive Weights:")
display(weights_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_df.tail(10))
Pegasos (poly) final model trained on entire train_poly. Bias (theta_0) = 0.622

Top 10 Positive Weights:
Feature Weight
27 x2*x9 2.558772
10 x8 1.371412
20 x1*x8 0.801088
6 x5 0.613006
12 x9 0.549907
8 x7 0.470546
0 x1 0.470503
16 x1*x2 0.188492
3 x2^2 0.186295
44 x3 0.146303
Top 10 Negative Weights:
Feature Weight
30 x4*x7 -0.046538
24 x2*x5 -0.059779
43 x9*x10 -0.068885
1 x1^2 -0.079423
37 x5*x10 -0.086601
45 x6 -0.118576
5 x4^2 -0.124102
17 x1*x4 -0.215107
4 x4 -0.392167
31 x4*x8 -0.818844

Weights inspection: the three models compared¶

In [36]:
#Perceptron Weights
weights_perc_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_perc_poly
}).sort_values(by="Weight", ascending=False)

print("=== Perceptron (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_perc_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_perc_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_perc_df.tail(10))


#Pegasos Weights
weights_pega_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_pega_poly
}).sort_values(by="Weight", ascending=False)

print("=== Pegasos (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_pega_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_pega_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_pega_df.tail(10))

#LogReg Weights
weights_logreg_df = pd.DataFrame({
    "Feature": expanded_feature_cols,
    "Weight": theta_logreg_poly
}).sort_values(by="Weight", ascending=False)

print("=== Logistic Regression (Polynomial) Weights ===")
print(f"Bias (theta_0) = {theta0_logreg_poly:.3f}")

print("\nTop 10 Positive Weights:")
display(weights_logreg_df.head(10))

print("\nTop 10 Negative Weights:")
display(weights_logreg_df.tail(10))
=== Perceptron (Polynomial) Weights ===
Bias (theta_0) = 2.300

Top 10 Positive Weights:
Feature Weight
27 x2*x9 108.472657
10 x8 57.616740
20 x1*x8 46.761335
12 x9 24.724363
6 x5 20.067918
0 x1 17.537309
8 x7 13.122208
44 x3 11.087068
41 x8*x9 7.061366
35 x5*x8 6.677310
Top 10 Negative Weights:
Feature Weight
9 x7^2 -3.733855
19 x1*x7 -5.428044
43 x9*x10 -5.533118
5 x4^2 -6.143356
45 x6 -6.267363
13 x9^2 -6.325809
18 x1*x5 -7.307161
17 x1*x4 -12.237412
4 x4 -21.615008
31 x4*x8 -29.930359
=== Pegasos (Polynomial) Weights ===
Bias (theta_0) = 1.222

Top 10 Positive Weights:
Feature Weight
27 x2*x9 5.118153
10 x8 2.733301
20 x1*x8 1.707763
12 x9 1.111045
6 x5 1.020537
8 x7 0.792118
0 x1 0.788783
44 x3 0.319826
23 x2*x4 0.226056
14 x10 0.161368
Top 10 Negative Weights:
Feature Weight
36 x5*x9 -0.086738
42 x8*x10 -0.115658
37 x5*x10 -0.131599
19 x1*x7 -0.137374
1 x1^2 -0.158771
5 x4^2 -0.277386
17 x1*x4 -0.383653
45 x6 -0.488428
4 x4 -0.727732
31 x4*x8 -1.621416
=== Logistic Regression (Polynomial) Weights ===
Bias (theta_0) = 0.105

Top 10 Positive Weights:
Feature Weight
27 x2*x9 0.885225
10 x8 0.562952
20 x1*x8 0.287563
8 x7 0.275661
0 x1 0.238675
6 x5 0.232460
12 x9 0.202625
3 x2^2 0.195755
16 x1*x2 0.169175
13 x9^2 0.164768
Top 10 Negative Weights:
Feature Weight
32 x4*x9 -0.023050
33 x4*x10 -0.024077
36 x5*x9 -0.030435
37 x5*x10 -0.036984
24 x2*x5 -0.053120
18 x1*x5 -0.071317
14 x10 -0.081209
45 x6 -0.090833
4 x4 -0.227507
31 x4*x8 -0.245479

Plot¶

In [37]:
models = ["Perceptron (poly)", "Pegasos (poly)", "Logistic (poly)"]
test_accuracies_poly = [test_acc_poly_perc, test_acc_poly_perc, test_acc_logreg_poly]

plt.figure(figsize=(7,4))
plt.bar(models, test_accuracies_poly, color=["skyblue","salmon","palegreen"])
plt.ylim([0,1])
plt.title("Final Test Accuracies on Polynomial-Expanded Features")
plt.ylabel("Accuracy")
for i, v in enumerate(test_accuracies_poly):
    plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.tight_layout()
plt.show()
No description has been provided for this image

Kernel Methods¶

Kernelized Perceptron with Gaussian and Polynomial Kernel¶

In [38]:
k = 5
target_col = "y"

Hyperparameter tuning¶

In [39]:
sigma_list = [0.1, 0.5, 1.0]
epochs_kperc_gauss = [3, 5]
best_acc_kperc_gauss = 0
best_sigma_kperc = None
best_ep_kperc = None

for sg in sigma_list:
    for ep in epochs_kperc_gauss:
        acc = k_fold_cross_validation(
            df=train_scaled,
            features=final_features,
            target=target_col,
            k=2,  # small because even with k= 5 the result is the same. The result shown below is calculated with k=5.
            random_state=42,
            classifier_func=lambda tr, vl, f, t: kernelized_perceptron_classifier_func(
                tr, vl, f, t,
                kernel_func=gaussian_kernel,
                kernel_params={"sigma": sg},
                epochs=ep
            )
        )
        if acc > best_acc_kperc_gauss:
            best_acc_kperc_gauss = acc
            best_sigma_kperc = sg
            best_ep_kperc = ep

print("\nKernel Perceptron (Gauss) best => sigma=%.3f, epochs=%d, CV=%.3f" % (best_sigma_kperc, best_ep_kperc, best_acc_kperc_gauss))
best_hparams_table["KernelPerceptronGauss"]["sigma"] = best_sigma_kperc
best_hparams_table["KernelPerceptronGauss"]["epochs"] = best_ep_kperc
best_hparams_table["KernelPerceptronGauss"]["cv_accuracy"] = best_acc_kperc_gauss
Kernel Perceptron (Gauss) best => sigma=1.000, epochs=5, CV=0.915

Gaussian Kernel Perceptron¶

Traning set evaluation

In [40]:
print("=== Kernelized Perceptron (Gaussian Kernel) ===")
sigma_value = 1.0   # Adjust as needed
epochs_gauss = 5    # Adjust as needed

accuracy_gauss = k_fold_cross_validation(
    df=train_scaled,  # or use a smaller subset if runtime is high
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
        tr_fold, vl_fold, f, t,
        kernel_func=gaussian_kernel,
        kernel_params={"sigma": sigma_value},
        epochs=epochs_gauss
    )
)
print(f"Gaussian Kernel: {k}-fold CV Accuracy = {accuracy_gauss:.3f}")
=== Kernelized Perceptron (Gaussian Kernel) ===
Gaussian Kernel: 5-fold CV Accuracy = 0.933

Polinomial Kernel Perceptron¶

Training Set Evaluation

In [41]:
print("\n=== Kernelized Perceptron (Polynomial Kernel) ===")
degree_value = 2    # e.g., quadratic
c_value = 1.0       # offset in (x^T y + c)^degree
epochs_poly = 5

accuracy_poly = k_fold_cross_validation(
    df=train_scaled, 
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_perceptron_classifier_func(
        tr_fold, vl_fold, f, t,
        kernel_func=polynomial_kernel,
        kernel_params={"degree": degree_value, "c": c_value},
        epochs=epochs_poly
    )
)
print(f"Polynomial Kernel (degree={degree_value}): {k}-fold CV Accuracy = {accuracy_poly:.3f}")
=== Kernelized Perceptron (Polynomial Kernel) ===
Polynomial Kernel (degree=2): 5-fold CV Accuracy = 0.925

Evaluation on Test set¶

Firstly, the Gaussian Kernel Perceptron

In [42]:
X_train = train_scaled[final_features].values
y_train = train_scaled[target_col].values
X_test = test_scaled[final_features].values
y_test = test_scaled[target_col].values

alpha_gauss = kernelized_perceptron_train(
    X_train, y_train,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_value},
    epochs=epochs_gauss
)

test_preds_gauss = kernelized_perceptron_predict(
    X_train, y_train,
    alpha_gauss,
    X_test,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_value}
)
test_acc_gauss = np.mean(test_preds_gauss == y_test)
print(f"Gaussian Kernel Perceptron Test Accuracy: {test_acc_gauss:.3f}")
Gaussian Kernel Perceptron Test Accuracy: 0.938

Then the polynomial kernel

In [43]:
alpha_poly = kernelized_perceptron_train(
    X_train, y_train,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_value, "c": c_value},
    epochs=epochs_poly
)

test_preds_poly = kernelized_perceptron_predict(
    X_train, y_train,
    alpha_poly,
    X_test,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_value, "c": c_value}
)
test_acc_poly = np.mean(test_preds_poly == y_test)
print(f"Polynomial (d={degree_value}) Kernel Perceptron Test Accuracy: {test_acc_poly:.3f}")
Polynomial (d=2) Kernel Perceptron Test Accuracy: 0.927

Kernelized Pegasos SVM¶

In [44]:
k = 5
target_col = "y"

Pegasos with Gaussian Kernel¶

In [45]:
lambda_val = 0.01
epochs_kpega = 5
sigma_val = 1.0

cv_acc_pegasos_gauss = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
        tr_fold, vl_fold,
        features=f, target=t,
        kernel_func=gaussian_kernel,
        kernel_params={"sigma": sigma_val},
        lambda_param=lambda_val,
        epochs=epochs_kpega
    )
)

print(f"Kernelized Pegasos (Gaussian, sigma={sigma_val}) {k}-fold CV accuracy: {cv_acc_pegasos_gauss:.3f}")
Kernelized Pegasos (Gaussian, sigma=1.0) 5-fold CV accuracy: 0.856

Now on the test set

In [46]:
alpha_gauss, T_gauss = kernelized_pegasos_train(
    X_train, y_train,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_val},
    lambda_param=lambda_val,
    epochs=epochs_kpega
)
test_preds_gauss = kernelized_pegasos_predict(
    X_train, y_train,
    alpha_gauss, T_gauss,
    X_test,
    kernel_func=gaussian_kernel,
    kernel_params={"sigma": sigma_val},
    lambda_param=lambda_val
)
test_acc_kernel_gauss_pega = (test_preds_gauss == y_test).mean()
print(f"Kernelized Pegasos (Gaussian) Test Accuracy: {test_acc_kernel_gauss_pega:.3f}")
Kernelized Pegasos (Gaussian) Test Accuracy: 0.846

Pegasos with Polynomial Kernel¶

In [47]:
degree_val = 2
c_val = 1.0

cv_acc_pegasos_poly = k_fold_cross_validation(
    df=train_scaled,
    features=final_features,
    target=target_col,
    k=k,
    random_state=42,
    classifier_func=lambda tr_fold, vl_fold, f, t: kernelized_pegasos_classifier_func(
        tr_fold, vl_fold,
        features=f, target=t,
        kernel_func=polynomial_kernel,
        kernel_params={"degree": degree_val, "c": c_val},
        lambda_param=lambda_val,
        epochs=epochs_kpega
    )
)

print(f"Kernelized Pegasos (Polynomial, deg={degree_val}, c={c_val}) {k}-fold CV accuracy: {cv_acc_pegasos_poly:.3f}")
Kernelized Pegasos (Polynomial, deg=2, c=1.0) 5-fold CV accuracy: 0.941
In [48]:
alpha_poly, T_poly = kernelized_pegasos_train(
    X_train, y_train,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_val, "c": c_val},
    lambda_param=lambda_val,
    epochs=epochs_kpega
)
test_preds_poly = kernelized_pegasos_predict(
    X_train, y_train,
    alpha_poly, T_poly,
    X_test,
    kernel_func=polynomial_kernel,
    kernel_params={"degree": degree_val, "c": c_val},
    lambda_param=lambda_val
)
test_acc_kernel_poly_pega = (test_preds_poly == y_test).mean()
print(f"Kernelized Pegasos (Polynomial) Test Accuracy: {test_acc_kernel_poly_pega:.3f}")
Kernelized Pegasos (Polynomial) Test Accuracy: 0.945

Final Data Visualization¶

In [49]:
final_hparams_df = []
for method, params in best_hparams_table.items():
    row = {"Method": method}
    for k, v in params.items():
        row[k] = v
    final_hparams_df.append(row)

final_hparams_df = pd.DataFrame(final_hparams_df)
print("\n=== Final Hyperparameter Table ===")
display(final_hparams_df)

# (Optional) Save final_hparams_df if you want
# final_hparams_df.to_csv("best_hyperparams.csv", index=False)
=== Final Hyperparameter Table ===
Method epochs eta cv_accuracy lambda sigma
0 Perceptron 10.0 0.1 0.646638 NaN NaN
1 Pegasos 20.0 NaN 0.724501 0.001 NaN
2 Logistic 10.0 1.0 0.717364 0.100 NaN
3 PerceptronPoly 10.0 0.1 0.925513 NaN NaN
4 PegasosPoly 20.0 NaN 0.945757 0.001 NaN
5 LogisticPoly 10.0 1.0 0.882041 0.100 NaN
6 KernelPerceptronGauss 5.0 NaN 0.915131 NaN 1.0
7 KernelPerceptronPoly NaN NaN NaN NaN NaN
8 KernelPegasosGauss NaN NaN NaN NaN NaN
9 KernelPegasosPoly NaN NaN NaN NaN NaN
In [50]:
# Example placeholders (DELETE or REPLACE with real computed values):
'''test_acc_perc_linear = 0.80
test_acc_poly_perc = 0.82
test_acc_perc_kernel_gauss = 0.84
test_acc_perc_kernel_poly = 0.83

test_acc_pegasos_linear = 0.81
test_acc_pegasos_poly = 0.83
test_acc_pegasos_kernel_gauss = 0.85
test_acc_pegasos_kernel_poly = 0.84
'''
# If you have logistic regression equivalents, define them too, e.g.:
'''test_acc_logreg_linear = 0.78
test_acc_logreg_poly = 0.80'''

# 1. We'll create a summary DataFrame
final_summary = []

# Perceptron
final_summary.append({
    "Model": "Perceptron",
    "Method": "Linear",
    "Test Accuracy": test_acc_perc
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_poly_perc
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Kernel (Gaussian)",
    "Test Accuracy": test_acc_gauss
})
final_summary.append({
    "Model": "Perceptron",
    "Method": "Kernel (Polynomial)",
    "Test Accuracy": test_acc_poly
})

# Pegasos
final_summary.append({
    "Model": "Pegasos",
    "Method": "Linear",
    "Test Accuracy": test_acc_pega
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_pega_poly
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Kernel (Gaussian)",
    "Test Accuracy": test_acc_kernel_gauss_pega
})
final_summary.append({
    "Model": "Pegasos",
    "Method": "Kernel (Polynomial)",
    "Test Accuracy": test_acc_kernel_poly_pega
})

# (Optional) Logistic
final_summary.append({
    "Model": "Logistic",
    "Method": "Linear",
    "Test Accuracy": test_acc_lr
})
final_summary.append({
    "Model": "Logistic",
    "Method": "Polynomial Expansion",
    "Test Accuracy": test_acc_logreg_poly
})

summary_df = pd.DataFrame(final_summary)
summary_df = summary_df.sort_values(by=["Model", "Method"]).reset_index(drop=True)
print("Final Test Accuracy Summary:")
display(summary_df)


# %%
# Barplot of final test accuracies by Model and Method
# Prepare data
models = summary_df["Model"].unique()
methods = summary_df["Method"].unique()

# Pivot the DataFrame for plotting
pivot_df = summary_df.pivot(index="Model", columns="Method", values="Test Accuracy")
display(pivot_df)

# Plot grouped bar chart
pivot_df.plot(kind="bar", figsize=(10,6))
plt.ylim([0, 1])
plt.title("Final Test Accuracies by Model and Method")
plt.ylabel("Accuracy")
plt.legend(title="Method", bbox_to_anchor=(1.0, 1.0))
plt.tight_layout()
plt.show()

# Single bar chart with all methods (Model x Method)
plt.figure(figsize=(8,6))
x_vals = range(len(summary_df))
acc_vals = summary_df["Test Accuracy"].values
labels = [f"{row.Model}\n{row.Method}" for idx, row in summary_df.iterrows()]

plt.bar(x_vals, acc_vals, color="skyblue")
for i, v in enumerate(acc_vals):
    plt.text(i, v+0.01, f"{v:.3f}", ha='center', fontweight='bold')
plt.ylim([0,1])
plt.xticks(x_vals, labels, rotation=45, ha="right")
plt.title("Final Test Accuracies (All Experiments)")
plt.tight_layout()
plt.show()
Final Test Accuracy Summary:
Model Method Test Accuracy
0 Logistic Linear 0.733950
1 Logistic Polynomial Expansion 0.882383
2 Pegasos Kernel (Gaussian) 0.845917
3 Pegasos Kernel (Polynomial) 0.945044
4 Pegasos Linear 0.732922
5 Pegasos Polynomial Expansion 0.949153
6 Perceptron Kernel (Gaussian) 0.937853
7 Perceptron Kernel (Polynomial) 0.927067
8 Perceptron Linear 0.670262
9 Perceptron Polynomial Expansion 0.923986
Method Kernel (Gaussian) Kernel (Polynomial) Linear Polynomial Expansion
Model
Logistic NaN NaN 0.733950 0.882383
Pegasos 0.845917 0.945044 0.732922 0.949153
Perceptron 0.937853 0.927067 0.670262 0.923986
No description has been provided for this image
No description has been provided for this image

save